Local tone mapping for high dynamic range images

ABSTRACT

A method of local tone mapping of a high dynamic range (HDR) image is provided that includes dividing a luminance image of the HDR image into overlapping blocks and computing a local tone curve for each block, computing a tone mapped value for each pixel of the luminance image as a weighted sum of values computed by applying local tone curves of neighboring blocks to the pixel value, computing a gain for each pixel as a ratio of the tone mapped value to the value of the pixel, and applying the gains to corresponding pixels in the HDR image. A weight for each value is computed based on distance from the pixel to the center point of the block having the local tone curve applied to compute the value and the intensity difference between the value of the pixel and the block mean pixel value.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. Provisional Patent ApplicationSer. No. 61/733,513 filed Dec. 5, 2012, which is incorporated herein byreference in its entirety. This application may be related to co-pendingU.S. patent application Ser. No. 14/098,230, filed Dec. 5, 2013, whichis incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the present invention generally relate to local tonemapping for high dynamic range images.

2. Description of the Related Art

The demand for higher dynamic range is a driving force for variousimaging applications such as, for example, security cameras andautomobile cameras. Imaging devices such as video cameras may beequipped with high dynamic range (HDR) sensors. Non-HDR cameras takephotographs with a limited exposure range, resulting in the loss ofdetail in bright or dark areas. Some HDR imaging devices may compensatefor this loss of detail by capturing two or more images at differentexposure levels and combining the images to produce images a broadertonal range than non-HDR devices. Merging multiple exposures preservesboth the saturated and the shadow regions and thus provides a higherdynamic range than a single exposure.

There are several known techniques for generating an HDR image (alsoreferred to as a wide dynamic range (WDR) image) from two or moreexposures. In one technique, the exposures may be spatially interleaved.In some techniques, the imaging system merges multiple exposures andprovides a native HDR Bayer image with a pixel depth ranging from 12 to20 bits. In some techniques, the imaging system captures multipletemporally spaced exposures and these exposures are merged to form anHDR image in the imaging device receiving the multiple exposures.Whether the imaging system generates the HDR image or the imaging devicegenerates the HDR image, tone mapping may need to be performed on theHDR image to permit processing of the HDR image in an imaging pipelinewith a lesser pixel bit depth, e.g., 10 to 12 bits.

SUMMARY

Embodiments of the invention relate to methods and apparatus for localtone mapping of high dynamic range images. In one aspect, a method oflocal tone mapping of a high dynamic range (HDR) image is provided thatincludes generating a luminance image of the HDR image, dividing theluminance image into overlapping blocks and computing a local tone curvefor each block, computing a tone mapped value for each pixel of theluminance image as a weighted sum of N values computed by applying localtone curves of N neighboring blocks of the pixel to a value of thepixel, wherein a weight for each value of the N values is computed basedon distance from the pixel to a center point of the block having thelocal tone curve applied to compute the value of the N values and anintensity difference between the value of the pixel and a mean pixelvalue of the block, wherein N is an integer, computing a gain for eachpixel of the luminance image as a ratio of the tone mapped value to thevalue of the pixel, and applying the computed gains to correspondingpixels in the HDR image to generate a tone-mapped HDR image.

In one aspect, an apparatus configured to perform local tone mapping ofa high dynamic range (HDR) image is provided that includes means forgenerating the HDR image, means for generating a luminance image of theHDR image, means for dividing the luminance image into overlappingblocks and computing a local tone curve for each block, means forcomputing a tone mapped value for each pixel of the luminance image as aweighted sum of N values computed by applying local tone curves of Nneighboring blocks of the pixel to a value of the pixel, wherein aweight for each value of the N values is computed based on distance fromthe pixel to a center point of the block having the local tone curveapplied to compute the value of the N values and an intensity differencebetween the value of the pixel and a mean pixel value of the block,wherein N is an integer, means for computing a gain for each pixel ofthe luminance image as a ratio of the tone mapped value to the value ofthe pixel, and means for applying the computed gains to correspondingpixels in the HDR image to generate a tone-mapped HDR image.

BRIEF DESCRIPTION OF THE DRAWINGS

Particular embodiments in accordance with the invention will now bedescribed, by way of example only, and with reference to theaccompanying drawings:

FIG. 1 is a block diagram of a digital system configured to performlocal tone mapping on high dynamic range (HDR) images;

FIG. 2 is a block diagram illustrating data flow of image processing inan embodiment of the digital system of FIG. 1;

FIGS. 3A and 3B are block diagrams illustrating data flow of imageprocessing in an embodiment of the digital system of FIG. 1;

FIG. 4 is a block diagram of a merge component in an embodiment of thedigital system of FIG. 1;

FIG. 5 is a flow diagram of a method for merging a long exposure imageand a short exposure image to generate an HDR image;

FIG. 6 is a flow diagram of a method for local tone mapping of an HDRimage;

FIG. 7 is a graph illustrating weight calculation for ghosting artifactreduction in a merged image;

FIG. 8 is an example illustrating local tone mapping;

FIGS. 9A and 9B are graphs illustrating generation of local tone curves;

FIGS. 10A, 10B, and 10C are examples illustrating computation ofdistance weights and pixel intensity weights for local tone mapping;

FIG. 11 is an example of applying local tone mapping to an HDR image;and

FIG. 12 is a graph illustrating derivation of parameter values fordetermining merging weights.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Specific embodiments of the invention will now be described in detailwith reference to the accompanying figures. Like elements in the variousfigures are denoted by like reference numerals for consistency. As usedherein, an image may be a single still picture of a scene or may be aframe in a video stream.

In general, embodiments of the invention provide for local tone mappingof high dynamic range (HDR) images. Some embodiments provide for mergingof long exposure and short exposure images to generate a merged (HDR)image. The local tone mapping includes computing a gain map from theluminance image of an HDR image using local tone curves and weightedbilateral interpolation. This gain map is applied to the Bayer channels(R, Gb, Gr, B) of the HDR image to generate the final tone-mapped HDRimage. Generation of a merged image includes computing weights forpixels of long and short exposure images and computing pixels values forthe merged image as the weighted sum of corresponding pixels in the longand short exposure images.

Unless otherwise specified, for simplicity of explanation, embodimentsare described herein in which pixels in an HDR image prior to tonemapping are assumed to be 16 bits. One of ordinary skill in the art,having benefit of the disclosure herein, will understand embodimentswith differing pixel bit depths, both before and after tone mapping.

FIG. 1 shows a digital system suitable for an embedded system (e.g., adigital camera) configured to perform local tone mapping of an HDR imageas described herein. In some embodiments, the digital system is alsoconfigured to merge a long exposure image and a short exposure image togenerate the HDR image. The digital system includes a DSP-based imagecoprocessor (ICP) 102, a RISC processor 104, and a video processingengine (VPE) 106 that may be configured to perform noise filtering asdescribed herein. The RISC processor 104 may be any suitably configuredRISC processor. The VPE 106 includes a configurable video processingfront-end (Video FE) 108 input interface used for video capture fromimaging peripherals such as image sensors, video decoders, etc., aconfigurable video processing back-end (Video BE) 110 output interfaceused for display devices such as SDTV displays, digital LCD panels, HDTVvideo encoders, etc, and memory interface 124 shared by the Video FE 108and the Video BE 110. The digital system also includes peripheralinterfaces 112 for various peripherals that may include a multi-mediacard, an audio serial port, a Universal Serial Bus (USB) controller, aserial port interface, etc.

The Video FE 108 includes an image signal processor (ISP) 116, and a 3Astatistics generator 118. The ISP 116 provides an interface to imagesensors and digital video sources. More specifically, the ISP 116 mayaccept raw image/video data from an HDR sensor module 126 (e.g., CMOS orCCD) and can accept YUV video data in numerous formats. The ISP 116 alsoincludes a parameterized image processing module with functionality togenerate image data in a color format (e.g., RGB) from raw sensor data.The ISP 116 is customizable for each sensor type and supports videoframe rates for preview displays of captured digital images and forvideo recording modes. The ISP 116 also includes, among otherfunctionality, an image resizer, statistics collection functionality,and a boundary signal calculator. The 3A module 118 includesfunctionality to support control loops for auto focus, auto whitebalance, and auto exposure by collecting metrics on the raw image datafrom the ISP 116 or external memory. In one or more embodiments, theVideo FE 108 is configured to perform local tone mapping of an HDR imageas described herein. In some embodiments, the Video FE 108 is configuredto generate the HDR image by merging a long exposure image and acorresponding short exposure image captured by the sensor module 126.

The Video BE 110 includes an on-screen display engine (OSD) 120 and avideo analog encoder (VAC) 122. The OSD engine 120 includesfunctionality to manage display data in various formats for severaldifferent types of hardware display windows and it also handlesgathering and blending of video data and display/bitmap data into asingle display window before providing the data to the VAC 122 in acolor space format (e.g., RGB, YUV, YCbCr). The VAC 122 includesfunctionality to take the display frame from the OSD engine 120 andformat it into the desired output format and output signals required tointerface to display devices. The VAC 122 may interface to compositeNTSC/PAL video devices, S-Video devices, digital LCD devices,high-definition video encoders, DVI/HDMI devices, etc.

The memory interface 124 functions as the primary source and sink tomodules in the Video FE 108 and the Video BE 110 that are requestingand/or transferring data to/from external memory. The memory interface124 includes read and write buffers and arbitration logic.

The ICP 102 includes functionality to perform the computationaloperations required for compression and other processing of capturedimages. The video compression standards supported may include, forexample, one or more of the JPEG standards, the MPEG standards, theH.26x standards, and the emerging HEVC standard. In one or moreembodiments, the ICP 102 may be configured to perform at least some ofthe computational operations of local tone mapping of an HDR image. Insome embodiments, the ICP 102 may be configured to perform at least someof the computational operations of merging a long exposure image and ashort exposure image to generate the HDR image.

In some embodiments, the HDR sensor module 126 is configured to captureHDR images of a scene and provides these images to the VPE 106 at asuitable frame rate, e.g., 30 frames per second (fps). In suchembodiments, the HDR sensor module 126 includes one or more suitableimaging sensors, e.g., CCD (charge-coupled device) or CMOS(complementary metal oxide semi-conductor) sensors. In some embodiments,the HDR sensor module is configured to capture a long exposure image anda short exposure image of a scene and provide these exposures to the VPE106 at a suitable frame rate, e.g., 60 fps. In such embodiments, the HDRsensor module 126 includes one or more suitable imaging sensors, e.g.,CCD or CMOS sensors.

In operation, in some embodiments, an HDR image of a scene is capturedby the HDR sensor module 124 and provided to the video FE 108. In suchembodiments, the Video FE de-compounds the HDR image for furtherprocessing. In some embodiments, a long exposure image and a shortexposure image of a scene are captured by the HDR sensor module 124 andprovided to the video FE 108. In the latter embodiments, the video FE108 merges the two images to form an HDR image of the scene. The mergingmay be performed as described herein. The video FE 108 converts the HDRimage to the input format needed to perform video compression. Prior tothe compression, local tone mapping as described herein may be appliedto the HDR image as part of processing the image in the image pipelineof the video FE 108. The video data generated by the video FE 108 isstored in the external memory. The video data is then encoded, i.e.,compressed. During the compression process, the video data is read fromthe external memory and the compression computations on this video dataare performed by the ICP 102. The resulting compressed video data isstored in the external memory. The compressed video data is then readfrom the external memory, decoded, and post-processed by the video BE110 to display the image/video sequence.

FIG. 2 is a block diagram illustrating data flow of image processing(the “image pipeline”) in an embodiment of the digital system of FIG. 1when the HDR sensor system 126 is configured to capture an HDR image ofa scene. FIGS. 3A and 3B are block diagrams illustrating data flow ofthe image pipeline in an embodiment of the digital system of FIG. 1 whenthe HDR sensor system 126 is configured to capture a long exposure imageand a short exposure image of a scene. One of ordinary skill in the artwill understand that similar functionality may also be present in otherdigital devices (e.g., a cell phone, tablet, PDA, etc.) capable ofcapturing HDR digital images and/or HDR digital video sequences. Whilemuch of the pipeline processing is shown in sequence, one of ordinaryskill in the art will understand that image processing is parallel innature and thus, each component in the pipeline does not necessarilyreceive an entire digital image before processing can begin. Thefunctionality of many of the image pipeline components is well-known andis not described in detail. Note that input images to the image pipelineare Bayer pattern images. Thus each pixel has only one of four colorcomponents—a red component denoted as R, a blue component denoted as B,or one of two green components denoted as Gr and Gb.

As shown in FIG. 2, the HDR image from the HDR sensor module 126 isreceived by a defect correction component 200. The defect correctioncomponent 200 is configured to correct the values of defective pixels inthe HDR image. As is well known, an image sensor may have some number ofdefective pixels which respond to light exposure differently than otherpixels due to factors such as manufacturing faults or operatingconditions. The correction may be performed using a look-up table (LUT)based technique or any other suitable technique. The defect correctedimage is provided to the black level adjustment component 202.

The black level adjustment component 202 is configured to set sensorblack to image black in the HDR image. That is, in order to optimize thedynamic range of the pixel values from the HDR sensor module 126, thepixels in the HDR image representing black are corrected since an imagesensor may record some non-zero current at these pixel locations. Notethat a black pixel should have a value of 0. The black level adjustmentcomponent 202 may adjust for this difference by subtracting offsets fromeach pixel value while clamping/clipping to zero to avoid a negativeresult. One simple way to calculate this adjustment is to take a pictureof complete black, e.g., by leaving on the lens cap or camera cover. Aseparate black level adjustment value may be used for each colorchannel. The adjusted HDR image is provided to the noise filtercomponent 204.

The noise filter component 204 is configured to remove various sourcesof noise in an HDR image, e.g., optical, electrical, digital and power,by averaging similar neighboring pixels. Typically, if the noise levelis high, more weight is given to the average of similar neighbors.Conversely, if the noise level is low, more weight is given to theoriginal pixel value. An Optical Electrical Conversion Function (OECF)chart captured using a uniform lighting source may be used to determinethe noise level for different intensities. The 12 uniform gray patcheson the OECF chart provide 12 power levels, which may then be used toarrange noise using either a linear or square-root model depending onthe sensor and gain (or ISO) level. The filtered HDR image is providedto the lens shading correction component 206.

The lens shading correction component 206 is configured to compensatethe HDR image for lens shading. Lens shading is the phenomenon that animage is bright in the center and decrease in brightness towards theedge of the field. Lens shading may be caused by factors such asirregularities in the optical properties of a lens associated with adigital image sensor or improper alignment between the Bayer colorfilter array and the lens. Any suitable technique for lens shadingcorrection may be used. For example, a gain may be applied on aper-pixel basis to compensate for any decrease in brightness. Theshading corrected HDR image is provided to the white balance component208.

The white balance component 208 is configured to adjust the white pixelsin an HDR digital image to compensate for color differences introducedby light sources, such as the differences between incandescent,fluorescent, natural light sources, XE strobe, and W-LED flash, as wellas mixed light conditions. That is, the illumination during therecording of a scene in a digital image may be different from theillumination when viewing the final digital image. This difference mayresult in a different color appearance that may be seen, for example, asthe bluish appearance of a face or the reddish appearance of the sky.Also, the sensitivity of each color channel varies such that grey orneutral colors may not be represented correctly. Any suitable whitebalance technique may be used. The white balanced HDR image is providedto both the 3A analysis component 210 (e.g., the 3A statistics generator118) and the tone mapping component 212.

The 3A analysis component 210 is configured to collect metrics from theHDR image for auto focus, auto white balance, and auto exposure ofsubsequent images. The tone mapping component 212 is configured toperform a method for local tone mapping on the HDR image as describedherein in reference to FIG. 6. The tone mapped image is provided to theremaining components 214 of the image pipeline for further processing togenerate the final HDR image. The further processing may include, forexample, RGB blending, gamma correction, conversion to the YCbCr colorspace, edge enhancement, contrast enhancement, and/or false chromasuppression.

As shown in FIG. 3A, the short exposure image from the HDR sensor module126 is processed by the defect correction component 300, the black leveladjustment component 302, the noise filter component 304, the lensshading component 306, and the white balance component 308 of the imagepipeline and stored in memory 310. The processed short exposure image isalso provided to the 3A analysis component 312. These components operateas described for similarly named components in FIG. 2.

As shown in FIG. 3B, the long exposure image from the HDR sensor module126 is processed by the defect correction component 300, the black leveladjustment component 302, the noise filter component 304, the lensshading component 306, and the white balance component 308 of the imagepipeline and is provided to the 3A analysis component 312. Thesecomponents operate as described for similarly named components in FIG.2. The processed long exposure image is also provided to the mergecomponent 314. The merge component 314 is configured to merge the longexposure image with the short exposure image stored in the memory 310 togenerate an HDR image. The operation of the merge component 314 isdescribed below in more detail in reference to the block diagram of FIG.4 and the method of FIG. 5. The merged (HDR) image is provided to thetone mapping component 316.

The tone mapping component 316 is configured to perform a method forlocal tone mapping on the HDR image as described herein in reference toFIG. 6. The tone mapped image is provided to the remaining components318 of the image pipeline for further processing to generate the finalHDR image. The further processing may include, for example, RGBblending, gamma correction, conversion to the YCbCr color space, edgeenhancement, contrast enhancement, and/or false chroma suppression.

FIG. 4 is a block diagram of the merge component 314 of FIG. 3B. Themerge component 314 includes a gain adjust component 400, a weightcomputation component 402, and a merge component 404. The inputs are along exposure image (processed as described above) and a correspondingshort exposure image (processed as described above). The exposure timesof the two images may set according to a ratio that is maintained as apower of 2. For example, the exposure time of the short exposure imagemay be two milliseconds (ms) and the exposure time of the long exposureimage may be 31 ms, with a ratio of ceil(31/2)=16. In another example,the exposure time of the short exposure image may be 4 ms and theexposure time of the long exposure image may be 29 ms, with a ratio ofceil(29/4)=8.

In general, the gain adjust component 404 adjusts the gain differencebetween the long exposure image and the short exposure image. The gainadjustment is described in more detail in reference to the method ofFIG. 5.

The weight computation component 402 determines the per-pixel weightsthat are to be applied to corresponding pixels of the long exposure andshort exposure images (after the gain adjustment) to generate the HDRimage. The per-pixel weights for the short exposure image and theper-pixel weights for the long exposure image are referred to herein as,respectively, Alpha_U(x,y) and Alpha_L(x,y). In addition, the weightcomputation component 402 determines motion adaptation weights to beused in the computation of the per-pixel weights for the long exposureimage. The motion adaptation weights, the computation of which may beper pixel or block-based, are applied to reduce ghosting artifacts thatmay be introduced in the merged image due to motion in the scene duringthe time between capture of the long exposure and short exposure images.Determination of the per-pixel weights and the motion adaptation weightsis described in more detail in reference to the method of FIG. 5.

The merge component 402 receives the per-pixel weights for the longexposure and short exposure images and the gain-adjusted images andcomputes each pixel M(x,y) of the merged image as perM(x,y)=(L(x,y)×Alpha_L(x,y)+U(x,y)×Alpha_U(x,y))where L(x,y) and U(x,y) are corresponding pixels in, respectively, thelong exposure image and the short exposure image, Alpha_L(x,y) is theweight for L(x,y), and Alpha_U(x,y) is the weight for U(x,y).

FIG. 5 is a flow diagram of a method for generating a merged image fromcorresponding long exposure and short exposure images that may beperformed by the merge component 314 of FIG. 3. Initially, the gaindifference between the two images is adjusted 500. The amount of gainadjustment for the long exposure image, gain_long, is computed asgain_long=(short exposure)/(long exposure)where short exposure is the exposure time of the short exposure imageand long exposure is the exposure time of the long exposure image. Toperform the gain adjustment, the pixels in the long exposure image aremultiplied by gain_long.

Per-pixel weights for pixels in the gain-adjusted long exposure imageand for pixels in the short exposure image are then computed 502. First,a per-pixel HDR (merging) weight, referred to as alpha(x,y) herein, iscomputed as per

${{alpha}\left( {x,y} \right)} = \left\{ \begin{matrix}1 & {{L\left( {x,y} \right)} > T} \\{{{af} \times {L\left( {x,y} \right)}} + {{bf} \times {L\left( {x,y} \right)}}} & {{L\left( {x,y} \right)} < T}\end{matrix} \right.$where x and y are pixel coordinates, L is a pixel value in the longexposure image, T is the saturation threshold, and the values of theparameters af and bf are derived based on T as described below.

The value of the saturation threshold T is the value of a maximumsaturated pixel after gain_long applied. Assuming the pixel depth of thelong exposure image is 16 bits, the value of a maximum saturated pixelis 2¹⁶−1=65535. When gain_long is applied to the long exposure image,the value of a maximum saturated pixel becomes 65535/gain_long. Forexample, if gain_long=¼, the value of a maximum saturated pixel becomes65535/4=16383. Thus, T is 16383.

FIG. 12 is a graph illustrating the derivation of the values of theparameters af and bf. In this graph, the x-axis is the pixel intensityin the long exposure image and the y axis is gain. The parameter V is atuning parameter for the derivation, the value of which may be anysuitable value between 0 and 1. The particular value of V isimplementation dependent and may be determined empirically. The equationfor the curve isaf ¹ +bf.At z=T/2, the equation for the point on the curve is

${{a\left( \frac{T}{2} \right)}^{2} + {b\left( \frac{T}{2} \right)}} = V$and at z=T, the equation for the point on the curve isaT ² +bT=1.Solving these equations,

${af} = \frac{\left( {2 - {4V}} \right)}{T^{2}}$gives the value of af, and

${bf} = \frac{\left( {{4V} - 1} \right)}{T}$gives the value of bf.

Motion adaptation weights, referred to as MAWeight herein, are alsocomputed for ghosting artifact reduction. In some embodiments, aper-pixel motion adaptation weight is computed. In some embodiments, amotion adaptation weight for each pixel is computed based on a block ofpixels around the pixel. Using per-pixel motion adaptation weights maybe more accurate but computationally expensive while using block-basedmotion adaption weights may be much faster but less accurate. The motionadaptation weight for a particular pixel location (x,y), MAWeight(x,y),is computed as per

${{MAWeight}\left( {x,y} \right)} = \left\{ \begin{matrix}1 & {{D\left( {x,y} \right)} < {D\; 1}} \\{1 - {\left( {{D\left( {x,y} \right)} - {D\; 1}} \right)*{slope}}} & {{D\; 1} \leq {D\left( {x,y} \right)}}\end{matrix} \right.$where D(x,y) is a computed delta between values of corresponding pixelsin the two images. FIG. 7 is a graph illustrating the weight calculationfor ghosting artifact reduction. The values of D1 and slope may bedetermined empirically. When there is motion in the scene, D(x,y)becomes larger, which reduces MAWeight, which then reduces the amount ofmerging. Less merging helps avoid ghosting artifacts that may occur dueto high motion.

For per-pixel ghosting artifact reduction, the value of D(x,y) iscomputed asD(x,y)=|U(x,y)−L(x,y)|where L(x,y) and U(x,y) are corresponding pixels in, respectively, thelong exposure image and the short exposure image. For block basedghosting artifact reduction, assuming a block size of N×N where N=2n+1(n=1, 2, 3, . . . ), the value of D(x,y) is computed asD(x,y)=max_(−n≦i≦n)(max_(−n≦j≦n) |U(x+i,y+j)−L(x+i,y+1)|).The value of n is implementation dependent and may be selected as atradeoff between computation speed and accuracy.

The per-pixel weights for the long exposure image and the short exposureimage are computed as perAlpha_L(x,y)=(1−alpha(x,y))*MAWeight(x,y)Alpha_U(x,y)=1−Alpha_L(x,y).

The merged (HDR) image is then generated 504 using the per-pixel weightsto combine corresponding pixel values from the gain-adapted longexposure image and the gain-adapted short exposure image. Morespecifically, each pixel M(x,y) of the merged image is computed as perM(x,y)=y)×Alpha_L(x,y)+U(x,y)×Alpha_U(x,y))where L(x,y) and U(x,y) are corresponding pixels in, respectively, thelong exposure image and the short exposure image, Alpha_L(x,y) is theweight for L(x,y), and Alpha_U(x,y) is the weight for U(x,y).

FIG. 6 is a flow diagram of a method for tone mapping of an HDR imagethat may be performed by the tone mapping component 212 of FIG. 2 or thetone mapping component 316 of FIG. 3. In general, the method takes the16-bit linear data of the HDR image and adaptively maps the data into asmaller number of bits based on the scene content in the image. Thenumber of bits to which each 16-bit pixel is mapped depends on the bitdepth of the ISP 116. For simple of explanation, mapping from 16-bitdata to 12-bit data is assumed. One of ordinary skill in the art willunderstand embodiments in which the smaller data size differs.

As shown in FIG. 6, initially a luminance image of the HDR image isgenerated 600. In some embodiments, the luminance image is down sampledto improve execution time. The amount of down sampling used isimplementation dependent. For example, a Bayer image of size 1280×736may be reduced to a luminance image of size 160×92.

White point adjustment is then performed 602 on the luminance image.White point adjustment in HDR images is important due to the widedynamic range. For example, for 16-bit pixel depths, the dynamic rangeis [0 65535] where black is 0 and white is 65535. If the values of thewhite pixels in the image are not close to true white, i.e., 65535, lowcontrast may result in the tone-mapped image. The white point adjustmentmay be performed as follows. First, a histogram of the pixel values inthe image is computed and the whitest pixel value, e.g., 65500, in theimage is determined from the histogram. Some small percentage of thewhiter pixels with values closest to the whitest pixel value, e.g.,2-5%, is then saturated to white, i.e., the values of these pixels arechanged to 65535. Further, after the saturation process, every pixel inthe image is scaled by the gain between the whitest original pixel valueand true white. The percentage of the whitest pixels to be saturated isan implementation choice.

For example, assume a luminance image of size 160×92 and that 2% of thewhiter pixels are to be saturated. The number of pixels in the image is14720 and 2% is approximately 294 pixels. Approximately 294 pixels areidentified for saturation by working backward from the identifiedwhitest pixel value, e.g., 65500, to identify pixels with the nextclosest value and the next closest value, etc. until the pixel count isapproximately 294. For example, suppose there are 100 pixels with avalue of 65550, 100 pixels with a next whiter value of 65500, and 100pixels with a next whiter value of 65495. The values of these 300 pixelsare saturated by setting them to be true white, i.e., 65535. Further,every pixel in the image is scaled by the gain between the whitest pixelvalue identified, 65500, and true white, 65535, i.e., every pixel ismultiplied by 65535/65500.

The luminance image is then divided 604 into overlapping blocks and ablock mean pixel value is computed 606 for each of the blocks. The blocksize and the amount of overlap between the blocks is implementationdependent and may be chosen empirically, for example, based on atradeoff between quality and performance. The block mean pixel value maybe computed in any suitable way, e.g., as the average of the pixelvalues in the block or as a weighted average of the pixel values in theblock. If a weighted average is used, the weight for each pixel may bebased on the inverse of pixel intensity difference between the centerpixel of the block and the pixel.

Local tone curves are then computed 608 for each block. These tonecurves may be of any suitable length. For simplicity of explanation, thelength of each local tone curve is assumed to be 256. One of ordinaryskill in the art will understand embodiments with differing tone curvelengths. The computation of a local tone curve of length 256 isillustrated in the graphs of FIGS. 9A and 9B. To compute a tone curvefor a block, three points on the curve are determined as illustrated inFIG. 9B. In this figure, CB is the block mean pixel value,TB=CB+delta_brightness is the target brightness, and LC and RC are,respectively, left and right contrast (i.e., the midpoint between 0 andCB and the midpoint between CB and 255, respectively). For purposes ofdetermining the three points of a tone curve, LC=0 and RC=255. Thus, thethree points of the local tone curve are [0 0], [CB TB] and [255, 255].The 256 points on the tone curve are constructed from these three pointsusing band limited interpolation and physicists' Hermite polynomials.

FIG. 9A illustrates the determination of the value of thedelta_brightness to be added to the block mean pixel value to determinethe target brightness TB. In this graph, the x axis is the block meanpixel value scaled from 16 bits to 8 bits and the y-axis isdelta-brightness (gain). As can be seen from this graph, for scaledblock mean pixel values less than the first knee point k₁, thedelta_brightness is the value of gain₁. For scaled block mean pixelvalues larger than the second knee point k₂, the delta_brightness is thevalue of gain₂. For scaled block mean pixel values between the two kneepoints, the value of delta-brightness is chosen along the gain curvebetween the two knee points. The determination of delta_brightnessvalues for particular block mean pixel values may be implemented as alookup table (LUT) of 255 values derived based on the values of k₁, k₂,gain₁, and gain₂. The knee points and the gains may have any suitablevalues, which may be selected empirically.

After the local tone curves are constructed for each block, a gain mapis computed 612 using the tone curves and weighted bilateralinterpolation. The resulting gain map contains a gain corresponding toeach pixel in the luminance image. The gain G(x,y) for a 16-bitluminance pixel X(x,y) in the luminance image with the value L iscomputed asG(x,y)=L _(out) /Lwhere L_(out) is computed as a weighted sum of applying the tone curvesof the four blocks having center points closest to X(x,y) to L. The fourblock centers closest to X(x,y) are referred to as the upper-left (UL)point, the upper-right (UR) point, the lower-left (LL) point, and thelower-right (LR) point herein. FIG. 8 is an example illustrating thelocations of the center points relative to X(x,y). Each of the smallcircles in the image is a center point of a block. For odd block sizes,the center point is the center pixel in the block. For even block sizes,the center point is between the four center pixels in the block.

More specifically, L_(out) is computed as perL _(out)=β^(UL) L _(out) ^(UL)+β^(UR) L _(out) ^(UR)+β^(LL) L _(out)^(LL)+β^(LR) L _(out) ^(LR)where L_(out) ^(NN) is the result of applying the tone curve of theblock containing the NN neighboring center point of X(x,y) and β^(NN) isa weight computed for this result, where NN={UL, UR, LL, LR}.

The value of L_(out) ^(NN) is computed as per

$L_{out}^{NN} = \frac{{{{LUT}\left\lbrack L_{floor} \right\rbrack}^{NN}W_{ceil}} + {{{LUT}\left\lbrack L_{ceil} \right\rbrack}^{NN}W_{floor}}}{W_{ceil} + W_{floor}}$where LUT^(NN) is the tone mapping lookup table for the block containingthe NN neighboring center point and L_(floor), L_(ceil), W_(floor), andW_(ceil) are computed as follows. The floor and ceiling values of L arecomputed as per

${L_{floor} = \frac{L}{256}};{0 \leq L_{floor} < 254}$L_(ceil) = L_(floor) + 1; 1 ≤ L_(floor) < 255and the floor and ceiling weights are computed as perW _(floor) =L−L _(floor)*256W _(ceil) =L _(ceil)*256−L.

The value of β^(NN) is computed as perβ^(NN)=α_(dw) ^(NN)α_(tw) ^(NN);0≦α_(dw) ^(NN),α_(tw) ^(NN)≦1where α_(dw) ^(NN) is a distance weight based on the distance from theNN neighboring center point to the pixel X(x,y) and α_(tw) ^(NN) is anintensity weight based on the difference in intensity between the meanpixel value of the block containing the NN neighboring center point andL.

Assuming that that the block size is odd and the center point is thecenter pixel of the block, the distance weight α_(dw) ^(NN) is computedas perα_(dw) ^(NN) =|x−x _(NN) ∥y−y _(NN)|/λwhere (x_(NN), y_(NN)) are the coordinates of the NN neighboring centerpoint and λ is a normalization factor to scale the distance weight to bebetween 0 and 1 inclusive. If the block size is even, the center pointis located between the four center pixels of the block, and the distanceweight is computed based on the distances from the pixel X(x,y) to eachof these four center pixels.

The intensity weight α_(tw) ^(NN) is computed as perα_(iw) ^(NN)=LUT[|L−L _(NN)|]where L_(NN) is the mean pixel value of the block containing the NNneighboring center point and the LUT is derived as per the graph of FIG.10C. FIGS. 10A and 10B are examples illustrating, respectively, thecomputation of the distance weights and the intensity weights forX(x,y). In the graph of FIG. 10C, the x axis is |L−L_(NN)| and they-axis is the intensity weight (gain). As can be seen from this graph,for values of |L−L_(NN)| less than the first knee point k₁, theintensity weight is the value of gain₁. For values of |L−L_(NN)| largerthan the second knee point k₂, the intensity weight is the value ofgain₂. For values of |L−L_(NN)| between the two knee points, theintensity weight is chosen along the gain curve between the two kneepoints. The determination of intensity weights for particular values of|L−L_(NN)| may be implemented as a lookup table (LUT) of 255 valuesderived based on the values of k₁, k₂, gain₁, and gain₂. The knee pointsand the gains may have any suitable values, which may be selectedempirically. In some embodiments, the values of the knee points and thegains are the same as those used to generate the LUT for determining thevalue of delta_brightness for particular block mean pixel values.

Referring again to FIG. 6, once the gain map is computed, the gains areapplied 612 to the four Bayer channels of the HDR image. In embodimentswhere the luminance image is downsampled, the gain map may be upsampledto the original HDR image resolution. The gains may be applied to thefour Bayer components of the corresponding pixels in the HDR image asperRbayer(x,y)=G(x,y)×Rbayer(x,y)Grbayer(x,y)=G(x,y)×Grbayer(x,y),Gbbayer(x,y)=G(x,y)×Gbbayer(x,y)Bbayer(x,y)=G(x,y)×Bbayer(x,y).

FIG. 11 is an example showing the effect of applying the above localtone mapping to an HDR image. The top image is the original 16-bit HDRimage converted to a 12-bit image without tone mapping and the bottomimage is the HDR image with local tone mapping applied. Note theimprovement in brightness and contrast when the local tone mapping isapplied.

Other Embodiments

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of thisdisclosure, will appreciate that other embodiments can be devised whichdo not depart from the scope of the invention as disclosed herein.

For example, embodiments have been described herein in which the localtone mapping applied to each pixel of an HDR image is based on fourneighboring blocks of the pixel. One of ordinary skill in the art willunderstand embodiments of the invention in which more or fewer blocksmay be used.

Embodiments of the methods described herein may be implemented inhardware, software, firmware, or any combination thereof. If completelyor partially implemented in software, the software may be executed inone or more processors, such as a microprocessor, application specificintegrated circuit (ASIC), field programmable gate array (FPGA), ordigital signal processor (DSP). The software instructions may beinitially stored in a computer-readable medium and loaded and executedin the processor. In some cases, the software instructions may also besold in a computer program product, which includes the computer-readablemedium and packaging materials for the computer-readable medium. In somecases, the software instructions may be distributed via removablecomputer readable media, via a transmission path from computer readablemedia on another digital system, etc. Examples of computer-readablemedia include non-writable storage media such as read-only memorydevices, writable storage media such as disks, flash memory, memory, ora combination thereof.

It is therefore contemplated that the appended claims will cover anysuch modifications of the embodiments as fall within the true scope ofthe invention.

What is claimed is:
 1. A method of local tone mapping of a high dynamicrange (HDR) image, the method comprising: generating, with one or moreprocessors, a luminance image of the HDR image; dividing, with the oneor more processors, the luminance image into overlapping blocks andcomputing a local tone curve for each block; computing, with the one ormore processors, a tone mapped value for each pixel of the luminanceimage as a weighted sum of N values computed by applying local tonecurves of N neighboring blocks of the pixel to a value of the pixel,wherein a weight for each value of the N values is computed based on adistance from the pixel to a center point of the block having the localtone curve applied to compute the value of the N values and an intensitydifference between the value of the pixel and a mean pixel value of theblock, wherein N is a positive integer; computing, with the one or moreprocessors, a gain for each pixel of the luminance image as a ratio ofthe tone mapped value to the value of the pixel; and applying, with theone or more processors, the computed gains to corresponding pixels inthe HDR image to generate a tone-mapped HDR image.
 2. The method ofclaim 1, further comprising performing white point adjustment on theluminance image.
 3. The method of claim 1, wherein N=4, and wherein theneighboring blocks are four blocks having center points closest to thepixel.
 4. The method of claim 1, further comprising: generating the HDRimage by merging a long exposure image and a short exposure image. 5.The method of claim 4, wherein merging a long exposure image and a shortexposure image comprises: computing a merging weight for each pixellocation of the long exposure image based on a pixel value of the pixellocation and a saturation threshold; and computing a pixel value foreach pixel location of the HDR image as a weighted sum of correspondingpixel values in the long exposure image and the short exposure image,wherein a weight applied to a pixel value of the pixel location of theshort exposure image and a weight applied to a pixel value of the pixellocation in the long exposure image are determined based on the mergingweight computed for the pixel location of the long exposure image andresponsive to motion in a scene of the long exposure image and the shortexposure image.
 6. The method of claim 5, wherein computing a pixelvalue further comprises: computing a motion adaptation weight for eachpixel location of the long exposure image; computing a weight for eachpixel location of the long exposure image based on the merging weightcomputed for the pixel location of the long exposure image and themotion adaptation weight computed for each pixel location of the longexposure image; and computing a weight for each pixel location of theshort exposure image by subtracting the weight computed for the pixellocation in the long exposure image from one.
 7. The method of claim 6,wherein computing a motion adaptation weight comprises: computing themotion adaptation weight for each pixel location of the long exposureimage based on a delta between a pixel value of the pixel location inthe long exposure image and a pixel value of the pixel location in theshort exposure image.
 8. The method of claim 6, wherein computing amotion adaptation weight comprises: computing the motion adaptationweight for each pixel location based on deltas between pixel values of ablock of pixel locations around the pixel location in the long exposureimage and pixels values of a corresponding block of pixel locationsaround the pixel location in the short exposure image.
 9. The method ofclaim 4, further comprising: adjusting gain difference between the longexposure image and the short exposure image.
 10. The method of claim 4,further comprising: performing black level adjustment on the longexposure image and the short exposure image; and performing whitebalancing on the long exposure image and the short exposure image. 11.An apparatus configured to perform local tone mapping of a high dynamicrange (HDR) image, the apparatus comprising: means for generating theHDR image; means for generating a luminance image of the HDR image;means for dividing the luminance image into overlapping blocks andcomputing a local tone curve for each block; means for computing a tonemapped value for each pixel of the luminance image as a weighted sum ofN values computed by applying local tone curves of N neighboring blocksof the pixel to a value of the pixel, wherein a weight for each value ofthe N values is computed based on a distance from the pixel to a centerpoint of the block having the local tone curve applied to compute thevalue of the N values and an intensity difference between the value ofthe pixel and a mean pixel value of the block, wherein N is an integer;means for computing a gain for each pixel of the luminance image as aratio of the tone mapped value to the value of the pixel; and means forapplying the computed gains to corresponding pixels in the HDR image togenerate a tone-mapped HDR image.
 12. The apparatus of claim 11, furthercomprising means for performing white point adjustment on the luminanceimage.
 13. The apparatus of claim 11, wherein N=4, and wherein theneighboring blocks are four blocks having center points closest to thepixel.
 14. The apparatus of claim 11, wherein the means for generatingthe HDR image generates the HDR image by merging a long exposure imageand a short exposure image.
 15. The apparatus of claim 14, wherein themeans for generating the HDR image merges the long exposure image andthe short exposure image by: computing a merging weight for each pixellocation of the long exposure image based on a pixel value of the pixellocation and a saturation threshold; and computing a pixel value foreach pixel location of the HDR image as a weighted sum of correspondingpixel values in the long exposure image and the short exposure image,wherein a weight applied to a pixel value of the pixel location of theshort exposure image and a weight applied to a pixel value of the pixellocation in the long exposure image are determined based on the mergingweight computed for the pixel location of the long exposure image andresponsive to motion in a scene of the long exposure image and the shortexposure image.
 16. The apparatus of claim 15, wherein computing a pixelvalue further comprises: computing a motion adaptation weight for eachpixel location of the long exposure image; computing a weight for eachpixel location of the long exposure image based on the merging weightcomputed for the pixel location of the long exposure image and themotion adaptation weight computed for each pixel location of the longexposure image; and computing a weight for each pixel location of theshort exposure image by subtracting the weight computed for the pixellocation in the long exposure image from one.
 17. The apparatus of claim16, wherein computing a motion adaptation weight comprises: computingthe motion adaptation weight for each pixel location of the longexposure image based on a delta between a pixel value of the pixellocation in the long exposure image and a pixel value of the pixellocation in the short exposure image.
 18. The apparatus of claim 16,wherein computing a motion adaptation weight comprises: computing themotion adaptation weight for each pixel location based on deltas betweenpixel values of a block of pixel locations around the pixel location inthe long exposure image and pixels values of a corresponding block ofpixel locations around the pixel location in the short exposure image.19. The apparatus of claim 14, further comprising: means for adjustinggain difference between the long exposure image and the short exposureimage.
 20. The apparatus of claim 14, further comprising: means forperforming black level adjustment on the long exposure image and theshort exposure image; and means for performing white balancing on thelong exposure image and the short exposure image.